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Systematic investigation of processes involved in 
needs analysis would improve current knowledge of the procedure and 
would bring analysts closer to the goal of generalized validity 
rather than situational validity. This article deals with two phases 
of the needs assessment process: selecting participants and defining 
needs and alternatives in terms of characteristics. A methodological 
approach for classifying participants is presented along with a 
method for determining intragroup consistency and agreement. Three 
methods of selecting participants are discussed: selection on the 
basis of affiliation with reference ! groups , on the basis of 
representing the population at large, and on the basis of demographic 
analysis that identifies significant groups. Actual choice of 
selection method has depended upon philosophical preference; for 
selection to be based upon more objective criteria a common unit of 
measurement must be identified. The author presents the concept of a 
numerical rating scale employing criteria for weighting intragroup 
consistency and agreement for participant selection. (MD) 
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Intragroup Inconsistency in 
L'ublic Policy Analysis 

Policymakers in various public service areas frequently employ 
rating scales to assess needs and to provide for alternatives. The 
result? of these efforts may be used in defining controversial issues, 
changing policies, and formulating program decision strategies. In 
the field of education, for example, needs analysis has for a long 
time been an important phase of planning and evaluating programs 
(Witkin, 1977). A need is the measure of the difference between an 
existing issue and the requirements for resolution of that issue. 
Since issues are controversial public problems for which relief is 
sought (Jones, 1977), all actions directed toward resolving an issue 
are or should be guided by policy. Therefore, policies on public 
education, housing, environment, and health, for example, may result 
indirectly or directly from a formal needs analysis. 

From this supposition the question arises, "Just where does the 
needs analysis fit into the total process of policy development?" As 
a beginning point, the five general phases in the policy process should 
be noted. These include: 

Issue awareness 

Problem definition 

Policy formulation 

Policy implementation 

Policy impact 

Conceptualization and measurement of needs usually transpire in 
the first three phases of the process outlined above. Although as 
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Policies are implemented and external impacts cause new issues, the 
last t.;o phases may also become components that require needs analysis. 

Even though many variations and semantic differences exist in 
the literature concerning needs analysis, all needs assessment 
techniques appear to have been derived from a basic schemata that 
actually consists of four steps (Miles, 1979). The first step involves 
establishing goals #hich reflect a desirable level of functioning for 
the program or organization under consideration. Secondly, evalua- 
tion of the program or organization's current level of functioning is 
required with respect to the pre-established goals. The third step 
consists of a discrepancy analysis based on any differences detected 
between the desired state (step 1) and the actual state (step 2) . 
Finally, the fourth step requires a prioritization of the expected 
outcomes for the purpose of administrative action. When these four 
steps are taken from the traditional needs assessment sotting and 
applied to the policy process, steps one and two deal with values and 
perceptions about issues instead of goals. Steps three and four 
involve a procedure for rating the magnitude of alternative needs. The 
ranking procedure used in the needs assessment may do more than provide 
a tentative comparison of alternatives. The strongest recommendation 
could be that certain alternatives should actually receive priority 
in further exploration and investigation (Quade, 1979). The latter 
statement is obviously directed toward phases three and four in the 
policy process. 

Information related to procedures for conducting a needs assessment 
in the applied setting is available from numerous conference proceedings 
(Travin, 1978, Southard, Note 1) and professional journals. One example 
includes a constructive criticism of needs assessment (Miles, 1979). 
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Witkin (Note 2), in another example Presented an evaluation of the 
needs assessment kits that were conunuricully available. For those 
electing practitioner oriented methods, Sarthory (7.977) provided 
guidelines for developing a needs assessment at the local level 
employing only those resources available within the local public 
service system. 

However, both the literature and conferences have not provided 
policymakers with adequate examples of empirical evidence concerning 
the utility or validity of the needs assessment in different policy- 
making or decision-making schema. Perhaps one reason for the lack 
of this important empirical evidence can be attributed to the specific 
validity paradigm most commonly employed tor assessing the needs 
analysis process. 

When the topic of validity (Cook and Scioli, 1979) or utility 
has been addressed in reference to the needs analysis procedures, the 
general method employed has been a concurrent validity paradigm. This 
model involves the process of correlating those outcomes obtained from 
needs assessment with external criteria. while the correlational 
procedure yields information relative to the covariation among out- 
comes and external criteria, it does not answer the question of why 
a relationship does or does not exist. Thus, the results of these 
correlational procedures are extremely limited. Relevant limitions 
include the inability to generalize to other settings or increase the 
current state of knowledge concerning whether or not needs assessment 
is worthwhile. Consequently, our knowledge of needs assessment 
techniques may not have increased appreciably in recent years. 

In view of the limitations associated with the concurrent validity 
paradigm, a shift in focus should emerge in th t; arena of needs assessment. 
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The impetus should be directed toward examining needs assessment as 
a process rather than an nsscssmont of .si t unt ionnl validity. By 
better understanding how needs assessment processes operate in a 
particular setting with a particular program, policy planners would 
be able to generalize their findings to other settings and other pro- 
grams. The advantages of a systematic investigation of processes 
involved in needs analysis would not only improve current scientific 
knowledge of the procedure but would also bring analysts a step closer 
to the goal of generalized validity rather than situational validity. 
An overview of the process would help clarify our position. 

Fundamentally, there are four general phases of the needs assess- 
ment process: (1) select the participants, (2) establish the issues 
or characteristics, (3) evaluate the characteristics, and (4) define 
the policy needs and program alternatives in terms of the character- 
istics. We shall deal with phases one and four in this article and 
present a methodological approach for clarifying participants. Next, 
a method for determining intragroup consistency and agreement shall be 
discussed . 

The choice of participants is important for two reasons. First, 
the participants play a fundamental part in the policy assessment pro- 
gram. Second, the participants provide quantitative and qualitative 
information concerning the programs under consideration, since this 
information is the very crux of the entire assessment process. Within 
this article we shall deal only with the quantitative dimension- 
Literature addressing the issue of participants has been devoted 
almost entirely to methods of participant selection. Sweigert (1971) 
has suggested that the participants to be chosen for the heeds assess- 
ment process should be selected on the basis of their affiliation with 
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reference groups. The basic idea of reference groups was also supported 
by Wilkin (1«J77) , but alio expanded Hie notion Lo include dissident 
groups, as well as those groups supportive o£ the issues or policies 
to be assessed. Witkin contended that by expanding the definition of 
reference groups to include both nonsupporters and supporters, those 
areas of greatest need within the program environment could be more 
readily identified. 

Another approach for selecting participants involves the selection 
of persons that represent the population at large. Kaufman and English 
(197^) recommend two different sampling procedures, either of which would 
yield participants representative of the population at large. One of 
their suggested procedures is that of a purely random sample in which 
each person within an identified area has an equal chance of being 
chosen. The other suggested procedure is that of a stratified random 
sample in which a proportional cross section of the population at large 
is identified and sampled. Of the two sampling procedures recommended, 
their preference is for the stratified approach. They contend that 
a priori sampling often forces the practitioner to think through, 
procedures and design considerations in more detail and eliminate 
possible sources of biases rather than attempting to 'doctor' a survey 
after the fact (Kaufman & English, 1979). 

Still another approach for selecting participants to take part in 
the needs assessment process was developed by Houston and Bain (19 79) . 
Their approach entails the development of a demographic analysis which 
covers a specified geographic area. The purpose of the demographic 
analysis is to identify significant social, political, and economic j 
groups within the specific geographic area. Once the significant groups 
have been identified, the formal leader of the groups would be solicited 
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as participants oi : the needs assessment process. This basic schemata 
idunt i Ty i ncj s i<jn i P i cn\\\r <jrouprs vin a doinoi iraph i c analysis war; also 
supported by Kaiser (1973) . However, he advocated the inclusion of 
both formal and informal leaders of the significant groups as partici- 
pants for the needs assessment process. 

These six specific methodologies may be subcjroupcd into threi.; 
separate categories. Within each of the three categories the basic 
selection paradigm for participants can be redefined to extend the basic 
scope of selection. The three classifications of the basic paradigms 
and their extensions are presented in Table I. An examination of the 
table indicates that Sweigert's notion of reference groups includes 
nonsupporters as well as supporters. Kaufman and English's simple 
random sampling was refined to a stratified sampling approach, A 
similar modification can also be observed for Houston and Bain's signifi 
cant- groups where Kaiser advocated the inclusion of informal and formal 
leaders . 

One major implication that can be drawn from the categorization 
shown in Table I is that any policymaker that initiates an assessment 
procedure and employs one of the basic selection classifications would 
obtain a discrete group of participants. Similarly, if one of the 
extended paradigms were selected, the group of participants obtained 
would differ from the group of participants obtained by any of the other 
' extended paradigms , while also differing from those selected by the 
basic paradigm. In the latter situation, the extended paradigm for the 
reference group and the significant group would include all those partici 
pants who would have been selected with the basic paradigm plus an 
increase of the additional participants identified by the expanded pro- 
cedure. 
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TABLE I 

THE CLASSIFICATION OF PARTICIPANT SELECTION PARADIGMS 



Classification 



Basic Paradigm 



Extented Paradiqm 



Reference Group 
Sampling 

Significant Group 



Supporters 
Random 

Formal Loaders 



Supporters and Non 
supporters 

Stratified Random 

Formal and In forma 
Leaders 
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Past practices reveal that the actual choice of selection method 
depends on philosophical preferences mther Uwin empirical evidence 
which supports the superiority of one approach over another approach. 
Obviously, the researcher should avoid any procedure that may tend to 
manipulate results when objectivity is the goal. However, within the 
domain of public service and education, the taxpaying public has 
changed the role of the policy planner and administrator. A shift from 
philosopher to pragmatist must transpire when expenditure of public 
funds is involved (Tarvin, 1978). In order to meet the requirements 
of this new role, the policy analyst will need empirical decision rules 
to justify the choice of a method implemented in participant selection. 
Increased substantiation will be of utmost importance when the partici- 
pant selection methods vary in respect to actual costs, work hours 
required, and types of results produced. 

Within the public policy process, consistency and agreement are 
imperative for sound planning. These signals of stability in opinion 
are appropriate during the issue awareness and throughout every addi- 
tional phase. Consistency among raters implies rank ordering that is 
the same although the weights may differ. Agreement is viewed as 
consensus regarding the weight or value of an issue area when judged 
by raters on a given scale. The desirability of obtaining agreement 
and consistency is not aimed at selecting biased samples of participants 
but is focused on obtaining participant groups that yield dependable 
information. 

Determining Intragroup Consistency and Agreement 
The previous section of this paper emphasized that the actual 
method which the policy analyst chooses for selecting participants 
to take part in a policy assessment will yield a distinct group of 
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participants.. Since each method of participant selection is idiosyn- 
cratic and the development of empirical decision rules requires some 
criteria that can be applied if any combination of methods is used, a 
common unit of measurement analysis must be identified for compara- 
tive purposes. 

One identifiable, common unit is a numerical rating scale (e.g. 1 
to 16-)- This type of scale measures the value of issues, programs, or 
policies to be assessed. The specific numerical input employed for 
comparing the different selection methods could be the participants' 
actual evaluation of an issue, the participants' projection of the 
desired level of functioning for the program policies, or the estimated 
modifications necessary for improvement of the program policies as 
derived from a relationship between the actual and desired character- 
istics. For the purpose of this article, the latter input will be 
employed for illustrative purposes. 

Evaluation of the participants* perceptions of improving program 
policies or need for change has previously been addressed only scantly 
in the literature under the general rubric of reliability. While the 
examination of these data via the reliability approach is certainly 
better than complete neglect, reliability does not provide all the 
information about the data that are needed by policy analysts and 
administrators who must make sound policy decisions. Reliability 
coefficients computed from the obtained data only provide partial 
insight concerning the participants' perceptions of improvement. In 
fact, the participants' perceptions can vary in a number of ways for 
which the reliability measure is insensitive. Therefore, any method- 
ology for evaluating the participants 1 perceptions must be sensitive 
to the different variations if the policy planner is to capitalize 
on the information available for decision making. 
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To visualize the extreme possibilities, we shall consider the 
hypot hotical data as shown in T.iblo ri. Thoro an? four d i f for on I: 
situations reflecting ways in which a group of participants could 
differ with respect to different program policies. The numerical data 
within each matrix indicate the level of need for improvement as 
defined by the participants' input. The higher the absolute value of 
a numerical input, the higher the need for improvement with respect 
to the given program policies. Each column vector of a matrix repre- 
sents a participants' assessed need for all the program policies under 
consideration. Each row vector of a matrix represents each program's 
level of need for improvement as assessed across all participants. 

An examination of the first example in Table II reveals a situa- 
tion which is optimal for the policy analyst conducting a needs assess- 
ment. One favorable aspect of the obtained data can be observed by an 
examination of the column vectors. Within each column vector, there is 
agreement between the relative standing of each program in respect to 
all other programs. Thus, each participant in the first example per- 
ceived the importance of the educational program policies as less than 
the importance of the public housing policies. Similarly, the importance 
of the public housing program was perceived as less than that of the 
environmental program policies. This factual degree of consistency 
denoted among the participants can be operationally defined in terms of 
intragroup consistency. 

Another favorable aspect of the obtained data in the first example 
can be observed by an examination of the row vectors. This examination 
reveals complete agreement by all participants in connection with the 
absolute need for each program. Wtiile the various programs are p 
ceived as having diverse levels of need, all participants noted an 
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TABLE II 



HYPOTHETICAL DATA DESCRIBING FOUK DIFFERENT SITUATIONS OBTAINED FROM A POLICY 



ASSESSMENT PROGRAM 



Program 
Policies 

on 

Education 

Public Housing 

Environmental 

Protection 

Agencies 

Public Health 



Example 1 
Participants 

12 3 4 



1111 

2 2 2 2 

3 3 3 3 

4 4 4 4 



Example 2 
Participants 

12 3 4 



1 5 9 13 

2 6 10 14 

3 7 11 15 

4 8 12 16 



Example 3 
Participants 

12 3 4 



12 2 1 

2 113 

3 4 3 2 

4 3 4 4 



Example 4 
Participants 

12 3 4 



12 3 4 

2 3 4 1 

3 4 2 1 

4 1 2 3 
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identical level of need per program. This singular level of need 
attached to a program among participants can be operationally defined 
as intragroup agreement. In the first example, the four participants 
exhibit perfect intragroup agreement, as well as intragroup consistency. 

An examination of the second example in Table II reveals both 
similarities and differences from the first example. Initially, the 
second example is similar to the first in that it exhibits a high 
degree of intragroup consistency. Similarly, participants in the 
second example perceive need of one program relative to the other 
programs the same across all column vectors. However, differences 
relative to intragroup agreement appear between the two examples. The 
level of absolute need associated with any one program shows low intra- 
group agreement in example two. That is, in the extreme situation, the 
first participant perceived the level of need for change in the educa- 
tional program policies as being much lower than the level of need for 
change as perceived by the fourth participant. Furthermore, the highest 
level of need for any program as perceived by the first participant is 
less than the lowest level of need as perceived by the other partici- 
pants. Thus, a low intragroup agreement level, is identified.- •,••'«. 

A problem for the policy analyst in the applied setting is shown 
in the second example. Since the data are characterized as having 
the low intragroup agreement, the analyst may have difficulty in 
determining the level of need for a program policy change. The uncertain 
need level would also weaken the justification for additional costs 
required to say exactly which policies are important via the assessment 
process. Therefore, the results of the needs assessment could be viewed 
by governing boards overseeing the evaluation process as a waste of time !: 
and effort. 



The third example in Table 11 presents a situation that differs 
somewhat from the previously discussed examples. The degree of intra- 
group agreement is relatively high in the third example. Within each 
of the row vectors all the participants basically perceived the need 
of the program as similar. However, the degree of intragroup consis- 
tency is shown to be low. The need of one program relative to the 
other programs is different for each participant as can be observed by 
an examination of the column vectors. 

In the third example we note a different problem for the policy 
analyst. The data can be described as having high intragroup agree- 
ment and low intragroup consistency. The researcher's difficulty would 

evolve around identification of the specific program with the greatest 

/ 

i)eed. Even though the actual level of need for all the programs is 
basically same across participants (high intragroup agreement), 

the order in which the programs are perceived is different (low intra- 
group consistency.) . In such a situation the policy analyst would be 
open to accusation by half the participants for asking advice without 
following the information received. This would be the situation 
regardless of the program area chosen for further work. 

The fourth example in Table II illustrates the worst of all 
possible situations. The data in the fourth example can be categorized 
as being low in regard to both intragroup consistency and intragroup 
agreement. Not only did the participants disagree about the need of 
one set of program policies as compared to the others, they also disa- 
greed about the absolute need for change in all areas. The actual 
identification and priority of any one program as compared to the other 
programs would be completely futile based upon the participants ' inputs 
in the fourth example. 
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The obvious conclusion that we cah draw from the discussion of 
tho <latn in Toblr IT i. 1 ; thai I ho first, c » >:;ini|>l • » ir. 11k* flr>:;ir<*c] i Lu#j~ 
tion. In the applied setting it is highly unlikely that the policy 
planner would ever encounter data which are perfect in terms of both 
intragroup consistency and intragroup agreement. It is also unlikely 
that he/she would encounter data that are extremely low in terms of 
both intragroup consistency and intragroup agreement as depicted in 
the fourth example of Table II. What does exist within the applied 
setting are data somewhere between the first and fourth examples. 

The nature of the variation among' the inputs affects interpretatio 
of the obtained data; thus, if the researcher is to optimize decisions 
about program policy changes, consideration must be given to both intra 
group agreement and intragroup consistency. Tn Oder to adequately 
utilize both of these concepts, the planner must operationalize them 
via a quantitative index. This index in its most elementary form 
consists of two correlation coefficients - one for consistency and one 
for agreement. To visualize the basis for the construct of the index, 
we should investigate the data matrices in Table II. These matrices 
shows rows that represent consistency and columns which exemplify agree 
ment. As the multiple correlation coefficients approach "I 11 greater 
consistency and agreement result. Autpcorrelation is the limiting 
factor in establishing a quantitative index as outlined here. Alterna- 
tive approaches to this problem will be suggested later in this article 

The advantages of quantifying the concepts of intragroup agreement 
and intragroup consistency are numerous. One advantage of employing 
empirical indices is the ease 1 with which large sauple data can be 
evaluated. The hypothetical examples employing only four participants 



evaluating Lour programs arc relatively easy La evaluate visually. 
However, in the field setting where the number of program policies as 
well as the number of participants are increased, an adequate visual 
evaluation of the data is impossible. Thus, the use of indices 
becomes not only advantageous but essential. 

Another advantage of employing empirical indices for evaluating 
the inputs obtained from participants in a policy study is that a priori 
decision rules can be established. The establishment of a priori rules 
for policy action permits the decision maker to justify giving certain 
responses more weight than others in the decision making process. 
Consequently, if some of the target groups do not show an acceptable 
degree of both intragroup consistency and agreement (concerning specific 
policy issues) then those I groups 1 inputs should receive relatively littl 
weight. By reducing the amount of weight given to groups without 
acceptable intragroup consistency and agreement, more meaningful action 
can be taken by policymakers than would be the case if all group inputs 
were treated equally. Not only would differential weighting produce 
more sound policy decisions, but the results obtained from the indices 
could provide planners with summary information relative to certain 
participant groups and how they view the need for change. Again, it 
should be emphasized that the criteria for group weighting are intra- 
group consistency and intragroup agreement rather than the positive or 
negative manner of viewing the programs. A sound decision concerning 
policy change can be made with cither high or low intragroup consistency 
and agreement, but not with information that is characterized as being 
low in intragroup consistency and high in intragroup agreement and 
vice versa. 
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What is a sound decision? By what quantitative criteria and 
whose set of values are decisions made? To beyin answering these 
questions we turn to the two components of consistency and agreement. 
Regarding the multiple correlation coefficients, an F ratio may be 
used to test whether either variable set is significantly different 
from 11 0" . Thus, the decision maker has a quantitative criterion as a 
basis for decision. Obviously, the values of the raters were the 
primary components of the quantitative index. Now the values of the 
decision maker are applied. Although his/her values and actions may 
be influenced by various pressure groups,, the decision is made. Prom 
this illustration we may infer that there is no such thing as a sound 
decision without noting that soundness is judged by the value sets of 
many individuals. This is, indeed, the case. 

Still another advantage of employing empirical indices is that the 
general state of knowledge concerning issue assessment can be enhanced 
through the accumulation of results which detail the nature of inputs by 
different groups to specific policy issues. Armed with historical 
evidence concerning the idiosyncrasies of different groups, the policy 
analyst may optimize future issue assessments by choosing those selec- 
tion strategies which would yield the greatest degree of consistency and 
agreement. The actual method chosen to select participants could be 
derived from the accumulated research employing indices. For example, 
the hypothetical indices in Table III reveal historial evidence of 
agreement and consistency among three participant selection paradigms 
and four public policy issue areas. To illustrate further, the best 
chance for the researcher to obtain consistency and agreement on educa- 
tional issues would be to select participants from the significant group 



TABLE III 



Hypothetical Indices of Consistency and Agreement 
By Classification and Issue 



Issue 
Arer. 



ducation 
ublic Housing 
nvironmental 
ublic Health 



Reference 
Srouo 



(.681) (.587) 
(.521) (. 499) 
(.479) (.566) 
(.433) (. 784) 



Sampling 



(.523) (.693) 
(.522) (.473) 
(. 833) (.699) 
(.824) (.822) 



Significant 
Group 

(.821) (.7314) 
(.523) (.503) 
(.528) (.498) 
(.627) (.774) 
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classification. On thu other hand, it duos not matter which classifica- 
tion is used to assess the public housing i«suo. 

Calculation of the indices for intragroup agreement and intra- 
group consistency is not a straightforward process with data obtained 
in the field setting. Since the number of persons and the number of 
programs in our example are equal and comprise an extremely small sample 
space, we have not defined mathematical terms or the formulae other 
than multiple correlation coefficients necessary to calculate the indices 
for complex situations. If an actual data set had been employed, rather 
than the hypothetical data set, the clarity of the examples would have 
been less, thus, obscuring the full potential these indices offer. Since 
the purpose of this article was to illustrate a novel way of analyzing 
issue assessment data regarding selecting participants, it was decided 
to error in the former direction rather than the latter. That is, the 
examples were oversimplified to make the point. 

In the applied setting, the fundamental issue to be resolved when 
calculating the indices is a determination of the level of measurement 
which characterizes the participants 1 assessed inputs. For the majority 
of issue analysis, the measurement decision will involve choosing 
between either an ordinal or assumed interval level of measurement. If 
the data are characterized as ordinal then the reader is referred. to 
Lawlis and Lu (1972) and Finn (1972) for the formulae of intragroup 
consistency and intragroup agreement, respectively. On the other hand, 
if. the data are characterized as an interval level**' of measurement, then 
the reader is referred to Bartko (1976) # Burdock, Fleiss, and Hardesty 
(1973) for formulae of intragroup consistency and intragroup agreement 
respectively. 
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Concilia i oh 

A review of the literature nddrenninq nocvln nsscssmoht revealed 
the popularity of tho technology as a tool in many situations. One 
researcher viewed needs assessment as an integral part of all policy 
planning (Kaufman, 1972). Other researchers suggested that needs 
assessment is absolutely fundamental to any evaluation (Scriven and 
Roth, 1978) . Still other researchers emphasized political and motiva- 
tional benefits (Ingersol, 1976) that can be derived from employing 
issue assessment. 

In view of the popularity and likelihood for continued use of this 
policy assessment technique, further research is needed to better 
understand the intricateness of this technology. The avenue of inves- 
tigation suggested in the current article entails viewing needs or 
issue assessment as a process. Four phases were identified, and the 
specific phases addressed here were participant selection and the 
evaluation of the participant's input in terms of consistency and 
agreement. 

Earlier studies approached the problem of participant selection 
via different selection paradigms, yet not even an empirical link 
exists between an issue area and the participants 1 input. Without 
evidence of input evaluation, the choice of one selection paradigm 
over another paradigm is at best questionable. However, through 
accumulated research employing the intragroup. indices suggested, 
practitioners would be able to choose the selection strategy expected 
to yield a high level of agreement and consistency. Also the numerical 
indices would . provide the practitioner with data that would justify 
differential weighting of certain groups 1 inputs. For example/ intra- 
group indices would reveal agreement and consistency among pressure 



groups and issue areas. From those, the policy researcher would have 
a measure of the degree of seriousness concerning needed policy changes • 
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